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Abstract 

At Goddard Space Flight Center (GSFC), we have chosen to use Capability Maturity 
Model Integrated (CMMI) 1 to guide our process improvement program. Projects at GSFC 
consist of complex systems of software and hardware that control satellites, operate 
ground systems, run instruments, manage databases and data and support scientific 
research. It is a challenge to launch a process improvement program that encompasses 
our diverse systems, yet is manageable in terms of cost effectiveness. 

In order to establish the best approach for improvement, our process improvement effort 
was divided into three phases: 

1 : Pilot projects 

2.: Staged implementation 

3: Sustainment and continual improvement 

During Phase 1 the focus of the activities was on a baselining process, using pre- 
appraisals in order to get a baseline for making a better cost and effort estimate for the 
improvement effort. Pilot pre-appraisals were conducted from different perspectives so 
different approaches for process implementation could be evaluated. Phase 1 also 
concentrated on establishing an improvement infrastructure and training of the 
improvement teams. 

At the time of this paper, three pilot appraisals have been completed. Our initial appraisal 
was performed in a flight software area, considering the flight software organization as 
the organization. The second appraisal was done from a project perspective, focusing on 
systems engineering and acquisition, and using the organization as GSFC. The final 
appraisal was in a ground support software area, again using GSFC as the organization. 

This paper will present our initial approach, lessons learned from all three pilots and the 
changes in our approach based on the lessons learned. 

1.0 Introduction: 

Software engineering is a core capability and key enabling technology necessary for the 
support of NASA’s missions. Ensuring the quality, safety and reliability of software is of 
paramount importance in achieving mission success. The exponential growth in the 
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scope, complexity, and importance of software is expected to continue, challenging our 
ability to manage it effectively. In order to meet these challenges, NASA has begun 
improvement initiatives in both software and systems engineering. 

For the NASA Software Initiative, a committee of representatives from each of the 
NASA Centers developed the NASA Software Initiative Implementation Plan to outline a 
comprehensive approach for improving software engineering to a quantifiable maturity 
level commensurate with the mission criticality of the software. The four strategies 
included in the NASA plan are: 

• To develop and implement Agency- wide and Center plans for continuous software 
process and product improvement in NASA and Contractor developed software; also 
to establish an infrastructure and measurement system to support improvement. 

• To improve safety, reliability, and quality of software products through the 
integration of sound software engineering principles and standards. 

• To provide input for research based on identified software problem areas and infuse 
research results. 

• To improve the software engineering knowledge base in NASA, and implement 
strategies for attracting and retaining software engineers. 

As a part of this plan, the use of either the Software Engineering Institute’s Capability 
Maturity Model (CMM) or the Capability Maturity Model Integrated (CMMI) as a 
measure of improvement was recommended. 

In response to the NASA Software Initiative Implementation Plan, Goddard Space Flight 
Center (GSFC) developed a software process improvement plan to address the four 
NASA strategies at Goddard. After careful consideration of the two recommended 
models, GSFC chose to use the CMMI model to measure its improvements. The 
remainder of this paper will focus on Goddard’s plan and the experiences gained during 
the early implementation of this plan. 

The Capability Maturity Model Integrated (CMMI) is a framework for process 
improvement containing a collection of best practices that can be expected of 
organizations producing quality systems and software. Concepts covered by this model 
include project management areas such as planning, monitoring and control, engineering 
areas such as technical solution, verification and validation, organizational areas (or 
process management areas) such as organizational training, and organizational focus, and 
support areas such configuration management and measurement. The staged 
representation of CMMI identifies five levels of maturity through which an organization 
evolves as it improves its processes. For GSFC, the initial plan was to achieve Level 3 
maturity on all projects determined to be mission critical or otherwise critical to GSFC’s 
business goals. 

In the systems engineering area, another NASA-wide team was formed to improve 
processes and best practices for systems engineering on projects and programs 
throughout the Agency. This group is also developing an improvement plan and is 
investigating the use of various models, such as the CMMI for use in their improvement 
program. At GSFC, the systems engineering group is working with the software people to 



evaluate the use of CMMI in the systems engineering areas. This pilot will help NASA 
with the choice of a model for systems engineering. 

2.0 Goddard Space Flight Center’s Improvement Plan Using CMMI 

The primary focus of the GSFC Process Implementation Plan is to improve the processes 
and practices in use at GSFC, using the CMMI as a measure of progress. The specific 
goals are to increase the percentage of on-time, within-cost software development 
projects, to increase productivity, and to reduce the post-delivery error rate after delivery. 
GSFC’s plan calls for a three phases of implementation as follows: 

• Phase 1 : A pilot project phase to baseline the processes and practices in use and to 
assess the implementation approach of the process improvement effort 

• Phase 2: Staged implementation of process improvement effort on all mission critical 
and mission support projects with the eventual target of raising the maturity levels, as 
measured against CMMI. 

• Phase 3: Maintain maturity achieved and support a continuous process improvement 
program 

2.1 Infrastructure for Improvement 

In order to manage the process improvementeffort three organizations were established: 
the Management Oversight Group, the Engineering Process Group, and the Asset 
Management Group. These groups will support the implementation of the process area 
activities and other improvements identified during the entire process improvement 
effort. These groups were formed during the initial phase of the implementation. The 
Groups and their interaction are described below: 

2.1.1 The Management Oversight Group 

The Management Oversight Group (MOG) is responsible for ensuring that the process 
improvement activities address GSFC needs, that GSFC institutional processes are 
defined, and that the stakeholder organizations are committed to using the established 
policies and procedures. During the initial phases, the MOG continually evaluates the 
progress and determines the potential source of needed resources. As CMMI 
implementation is rolled out to GSFC projects, the MOG will determine the schedule for 
projects and programs participating in the process improvement effort. 

2.1.2 The Engineering Process Group 

The Engineering Process Group (EPG) is the focal point for software process 
improvement across GSFC, and its responsibilities have recently been expanded to 
include systems engineering. The EPG maintains the overall view of current efforts and 
facilitates those efforts on a continuing basis. The EPG develops and maintains expertise 
in process definition, organizational change, and technology related to improving and 
ensuring software/systems quality. The EPG also facilitates deployment of improved 
software/systems development activities by working closely with the projects to enhance 
their understanding and application of any new process and technology improvements. 
This is critical to the success of the process improvement implementation. EPG members 
foster collaboration among everyone at GSFC involved with software/systems 
development process improvement or application of the practice. The EPG is staffed by 



software/systems process engineering experts and by software/systems engineers with 
practitioner experience representing all directorates across GSFC. 
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2.1.3 The Asset Management Group 

The Asset Management Group (AMG) supports the EPG and focuses on the 
establishment and maintenance of the infrastructure elements necessary to support a 
continuous process improvement effort. It is chartered with maintaining a profile of 
GSFC software processes and products, and with the collection of information that will 
maximize the return on GSFC investment in software intellectual property. These 
activities will continue throughout the process improvement activities. To this end the 
AMG is responsible for software knowledge management, the on-line deployment of 
GSFC documented software policies and procedures, and the provision of software 
engineering tools that support GSFC software engineering practices. Although currently 
focused on the software process improvement effort, there are plans to expand the AMG 
to include systems engineering next year 


3.0 Phase 1 Activities 

Currently GSFC is working to complete its Phase 1 activities and to begin more detailed 
planning for Phase 2. As mentioned previously, several pilot areas were chosen to be 
representative of software developed at GSFC. Then these pilot areas were evaluated 
against the staged CMMI model using SEI-certified appraisers during pre-appraisals. 

The objectives of using pilot projects in Phase 1 of the implementation were to: 

• Understand the resources (i.e., resources defined primarily as personnel time) needed 
to implement process areas in the CMMI maturity levels two and three, 

• Understand the comprehensiveness and interaction between the process areas, 

• Be aware of critical issues and potential problems with process improvement in 
relation to the GSFC culture, 

• Be aware of major issues and the impact of metrics collection required for CMMI, 



• Establish the baseline of current costs and quality of software developed. 

Originally, four pilot areas were planned. To date, three pilot areas have been appraised, 
and the fourth has been deferred due to the lack of available projects meeting the criteria 
for the fourth pilot. 

3.1 Selection of Pilot Areas 

At GSFC, a software project can be described in its simplest form as composed of flight 
software, ground software, and instruments. The flight and ground software along with 
the instruments will be viewed as suppliers to the project to simplify the application of 
the CMMI model. Therefore, there are four types of components that need to be assessed 
against CMMI: the project itself (including systems engineering and acquisition), the 
flight software, the ground software, and each instrument. It was recommended that there 
be four pilot areas, one from each component area. For Phase 1, one component project 
from each pilot area was chosen for improvement activities following the appraisal. 

This is shown below by the arrows. 



Pilot Project Implementation Selection 

The following criteria were used to select projects in each pilot area: 

• Project is already established so that there are existing procedures to evaluate. 

• Project management is stable, willing to support participation in the process 
improvement effort, and has the necessary funding available. 

• Project is comprised of enough project members to ensure that all process areas can 
be evaluated and improved. 

• Project has visibility within GSFC to ensure appropriate emphasis on the process 
improvement effort. 

3.2 The First Appraisal 

Our initial appraisal was performed on the Flight Software area, using one completed 
project with all documentation in place and one project nearing the preliminary design 
review point. For this appraisal, the organization was considered the flight software 
organization, and few interviews included people outside of this organization. 

Project participants were given a three-day CMMI class, but had very little other 
preparation for the pre-appraisal. No attempt was made to map their documentation to 




CMMI process areas or to identify alternative practices that might be in place before the 
appraisal. Most of the flight software participants were on very tight schedules and every 
attempt was made to minimize their preparation and interview time during the appraisal 
process. 

A team of six appraisers was used; three were EPG members and three were SEI-certified 
appraisers from outside GSFC. The appraisal team divided the process areas and each 
took the lead on interview questions in their area. The appraisal took four days with the 
majority of the interviews on two days and the remaining two days devoted to document 
review. Although less formal, a SCAMPI-type of approach was used, where findings are 
the result of team consensus, supported by multiple data points from multiple sessions. A 
findings briefing was given on the last day describing the weaknesses identified in each 
process area by specific practice. A draft findings presentation was not given which 
resulted in problems during the final findings briefing. Practices in both Levels 2 and 3 
were evaluated, but no maturity level rating was given. 

Following the appraisal, the EPG team members who had participated in the appraisal 
met with the flight software manager to discuss the weaknesses and her priorities for 
addressing them. Then the EPG determined what existing assets and resources could be 
applied in those areas and made an improvement proposal to the manager. Based on the 
improvement proposal and the flight software manager’s priorities, five process action 
teams were formed to address weak areas. 

3.3 The Second Appraisal 

The second appraisal was done from the project perspective, focusing on systems 
engineering and looking at the software components as acquisitions. Three major 
spacecraft projects were chosen to participate. One was still in the formulation stage, 
nearing a systems requirements review. The second was a major project in mid- 
development, with about two years remaining until launch and the third was a project 
within a program developing a series of “turn-key” spacecraft. For this appraisal, the 
organization was considered to be GSFC, since GSFC is a matrix organization with 
personnel from different functional groups assigned to each of the projects. 

Project participants were given an overview of CMMI, but not the three-day class. 
Although there was still a desire to minimize the amount of the project personnel time, 
participants were given lists of example types of documentation and sample types of 
questions. 

Again, a team of six appraisers was used, with three SEI-certified Lead appraisers and 
three EPG members. This time, the EPG members of the appraisal team included one 
systems engineer, one quality assurance person and one software person. The SEI- 
certified Lead appraiser asked the majority of the interview questions. The same 
methodology was followed as in the first appraisal, except that more emphasis was placed 
on interviews and less on documentation. The appraisal took five days, with most of three 
days devoted to interviews. The fourth day was used for document review and a draft 
findings presentation. The final briefing was presented in terms of weaknesses and 



strengths for the goals of the process areas in Levels 2 and 3. No maturity ratings were 
determined. 

3.4 The Third Appraisal 

The third appraisal was similar to the first in the choice of projects, since the projects 
chosen were subsystems of larger projects that were being managed relatively 
independently of their larger project. Two projects were chosen, both being developed in 
a ground support system software organization. One project was completed, and the 
development team was just beginning a very similar project. The second project was in a 
testing phase. For this appraisal, the organization was again considered to be GSFC. The 
larger project was considered for the areas of support it provided. 

A new appraisal methodology was used for this appraisal. During the course of the year 
that we had been trying to do pilot appraisals, the appraisal methodology for this type of 
“quick-look” appraisal had been evolving at the Software Engineering Institute and 
among the authorized appraisers. A new tool called a Process Implementation Indicator 
Document (PHD) was introduced for this appraisal. This document listed all the expected 
practices in the model and provided space to record any evidence of the practice in boxes 
marked “direct work product”, “indirect work product” or “affirmation.” 

In order to use the tool, the appraisal team, consisting of two SEI-authorized appraisers 
and four GSFC EPG members, was given a three-day “team training” to learn how to use 
the PUDS. This occurred about a month before the actual appraisal. The internal appraisal 
team members met with the leads of the participating projects and gathered what they felt 
would be appropriate documentation to begin populating the PHD. Then the GSFC EPG 
team members and the SEI-authorized appraisers spent another week working to fill in 
the PIID information, using the documentation gathered from the team leads. No 
interviews were done during this week of preparation for the appraisal. 

Again, with this appraisal, there was a strong desire to minimize the time of the project 
people spent on the appraisal. Project participants were again given an overview of 
CMMI, but were not involved in the initial efforts to populate the PUDS. During the 
actual appraisal week, the project people were interviewed to verify what had been 
entered in the PUDS and to leam about anything that had not been collected initially. 
Again a draft findings briefing was held and then a final briefing of the strengths and 
weaknesses on the fifth day. The process areas at levels 2 and 3 were examined, but due 
to the time constraints, most of the process areas with an organizational focus were not 
evaluated. 

4.0 Lessons Learned to Date 

To date, initial evaluations of the activities and the processes used during Phase 1 are 
being summarized and have resulted in lessons learned for future activities. Some lessons 
learned are GSFC-specific, but many others may be useful to other organizations 
beginning process improvement. The lessons presented here relate to training, choice of 
organization for improvement, management support, and pre-appraisal lessons. 



4.1 Definition of the “organization” can be difficult 

For the first appraisal, GSFC defined the “organization” to be appraised as the line 
organization that contained developers/engineers producing like types of component 
software or services. Examples of these types of organizations would be “flight software 
organization”, “ground software organization”, “systems engineering organization”. 

Since GSFC uses a matrix organization, groups from each of these types of organizations 
are assigned to support each spacecraft project. The line management organization is 
responsible for the training and detailed process activities of these groups, but all project 
management activities (requirements, schedules, budgets, interface coordination, etc.) are 
performed by the spacecraft project organization. For the second and third appraisals, the 
“organization” was defined as GSFC, a much larger entity, which contains both the 
spacecraft project organizations as well as the smaller component organizations. 

The advantage of defining the organization using the component types of organizations is 
that the scope you are dealing with is much smaller. The primary disadvantage in a 
matrixed organization such as GSFC is that many of the process areas are addressed at 
GSFC by other component organizations, making it very difficult to determine whether 
the appropriate practices are being performed without looking in detail at the other 
supporting organizations. 

4.2 Management support is key to finding co-operative pilots 

Typical projects at GSFC have deadline-driven schedules and a budget that is usually less 
than desirable. Thus project managers are not anxious to engage in any types of activities 
such as appraisals or classes that will not directly assist them with their primary goal — 
that of delivering an operational spacecraft system. Initially, the EPG members spoke 
with many of the project managers about participating in the Phase 1 pilot areas. Very 
few were willing to participate in the appraisals. 

In order to obtain participants for the second set of appraisals, the EPG enlisted 
assistance from the MOG. One of the members of the MOG accompanied the EPG 
member to present the request for participation to the projects. Using this approach, every 
project contacted agreed to participate. Similarly, in the third appraisal, the line manager 
of the component organization was very supportive, and the projects chosen in that 
organization were very cooperative. 

4.3 Pre-appraisals by an outside group provide a good starting point 

Many organizations beginning process improvement start by working on the areas they 
feel are weak, without having the benefit of external guidance. GSFC chose to have SEI- 
certified appraisers outside NASA help them with their initial benchmarking against the 
CMMI model. In retrospect, there are many advantages to this approach. 

First, in one short week, we were able to amass a tremendous amount of information on 
exactly how activities were being performed in different areas and what documentation 
was being used. By using multiple projects in our “pilot areas”, we also got a good view 



of the consistency of activities across projects. This baseline information was very useful 
in providing a benchmark of comparison with industry standards for senior management 
to evaluate and determine what types of improvements they would like to sponsor. 

The use of appraisers outside NASA enabled the EPG to get a good view of exactly how 
the existing GSFC processes and documentation would be viewed in a more formal 
appraisal. In some cases, the documentation reviewed was considered sufficient by the 
outside appraisers, when the EPG would have considered that additional documentation 
was necessary. The use of outside appraisers also gave a more official flavor to the pre- 
appraisals, which encouraged the participating projects to assure that the appropriate 
people would be available for interviews. 

Finally, the major benefit of using the pre-appraisals with certified Lead appraisers was 
training it provided to the EPG members participating in the appraisals. EPG members 
now have a very clear idea of exactly how a SCAMPI appraisal would be conducted, and 
a very good idea of how the CMMI model would be applied to activities at GSFC. In 
addition, the process of going through an appraisal is very helpful in becoming more 
familiar with the CMMI model and its interpretation. The model is very large and 
complicated, and it takes time to learn it well. The appraisal process provides a practical 
way to increase model knowledge in a short time period. 

The major disadvantage of spending a significant percentage of our available process 
improvement effort on the pre-appraisals was that very few resources were available for 
actual improvements in the areas of weaknesses discovered. This resulted in very slow 
improvements on the actions identified for the pilot project from the first appraisal and 
very slow feedback on action planning for the pilot project from the second project. A 
recommendation for others using this pre-appraisal approach is to assure that sufficient 
staffing is available to assist with improvements immediately following the pre- 
appraisals. 

4.4 Determining an Accurate Cost and Effort Estimate for Process Improvement is 
Difficult 

The series of pre-appraisals provided enough information to determine the gaps between 
our current capability and our desired capability and gave us the necessary knowledge to 
develop a good work breakdown structure of tasks necessary for the improvement. We 
were also able to get fairly a good count of items like amount of process documentation 
to be developed. Our initial plan was to estimate the number of tasks to be done in the 
work breakdown structure and the cost of these tasks for a better total cost estimate for 
improvement. While we were able to do this, our experience with the improvement 
efforts we initiated demonstrated a wide variability in cost for seemingly similar tasks. A 
larger sample of improvement activities might have given us more confidence in the 
accuracy of our cost estimates. 



4.5 Pre-Appraisal Lessons Learned 

In comparing the process used for performing the three pre-appraisals, we were able to 
draw the following conclusions: 

• Allow enough preparation time: It takes time to schedule interviews, handle logistics 
and plan for an appraisal. We spent at least several weeks in preparation for each of 
the first two pre-appraisals and about a month for the third appraisal. 

• Preparation of participants yields better results. During the second appraisal, when we 
helped participants identify documentation and provided sample types of questions, 
we were able to get more accurate results. In the first appraisal, some participants 
neglected to mention some critical activities that were being performed since they 
were not very familiar with the CMMI model. 

• The CMMI model worked well to evaluate systems engineering activities. The focus 
of the second appraisal was systems engineering with a large amount of time devoted 
to baselining system engineering activities. Both the GSFC personnel and an outside 
systems engineering expert felt that the CMMI model provided a reasonable 
measurement tool for benchmarking these activities. 

• Do a draft findings briefing. A draft findings briefing helps identify any 
misunderstandings or gaps in information that might exist after the interview periods. 

• The use of the PHD to capture information during the pre-appraisal provided the most 
information for improvement activities following the appraisal. Using the PHD, it was 
much easier to go back through and see exactly what types of practices were in place 
for each project and what the artifacts of the practices were. However, the use of the 
PIIDs caused the pre-appraisals to take longer to perform. 

• Follow-up is important. When projects have agreed to participate in appraisals and 
process improvement activities, they need to receive the feedback from the appraisal 
in a timely fashion, as well as information on follow-on activities. 

4.6 Provide the right training for the right people 

One group of project participants was given a three-day CMMI class and the other two 
groups were given a CMMI overview. We recommend the overview training for project 
participants. The knowledge the project participants really needed was a good knowledge 
of the processes used on their projects and the higher levels of process documentation 
from which their project documentation was derived. The three-day CMMI class was a 
significant time commitment for most of the project people and they felt overwhelmed 
with the volume of information. Many attendees thought all process area practices needed 
to be satisfied by their local organization and overlooked the contributions of the 
supporting organizations. 

All EPG members received the three-day CMMI training. The detailed knowledge of 
CMMI has provided many benefits to the EPG team, allowing them to assist in mapping 
existing practices to those of CMMI and to identify alternative practices that are being 
used. For the third appraisal, the EPG appraisal team members had completed the 
Intermediate CMMI class, which allowed them take a first cut at filling out the appraisal 
PIID forms, thus taking much of the time burden off the projects for the pre-appraisal. 



5.0 Applying Our Lessons For Phase 2 

As a result of the lessons we have learned during our pilot phase, we are making several 
changes in our approach for Phase 2. All the pre-appraisals were performed using the 
Staged Representation of the CMMI model, and we have decided to adopt the Continuous 
Model for Phase 2. This allows us to choose the areas for improvement that are the most 
important to the organization since our resources are too limited to improve everything at 
once. We also plan to focus on a smaller, more critical portion of the whole organization 
so that we will be able to apply enough resources to make progress quickly. This will also 
provide the necessary “early success” needed to ensure continued management support. 

Conclusion 

As GSFC completes its first year of process improvement and the end of Phase 1 
activities, we are finding that the CMMI model is a useful way to measure an initial 
capability state of an organization and to identify improvement opportunities. The initial 
activities of the year have assisted us in establishing a good workable infrastructure for 
improvement, training our process improvement team members, baselining our current 
capabilities, and gaining a better understanding of the cost and effort necessary to 
continue our process improvement efforts. All of this information, as well as our lessons 
learned during the past year will facilitate our planning for a successful transition into 
Phase 2: The phased inclusion of many more projects. 



